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ABSTRACT 



This paper focuses on the experience of the Policy Education 
Centre on Assistance to Transition in providing monitoring and evaluation 
services to "Support for Improvement in Governance and Management in Central 
and Eastern European Countries" (SIGMA) , a joint project of the Organization 
for Economic Cooperation and Development and the Poland and Hungary 
Assistance in Restructuring Economies program (PHARE) . Part 1 of the paper 
describes the challenges of the experience in terms of context, program 
design, and performance monitoring. The SIGMA program was intentionally 
nonprescriptive , offering a menu of options about the types, speed, and 
sequencing of reforms, and the paths clients followed varied widely. Part 2 
of the paper provides descriptions of how certain problems were addressed, 
such as the design and implementation of the monitoring and evaluation 
strategy, the process of establishing performance measurements, and the 
development of tools for data collection. The challenge to evaluators was to 
build suitable performance standards for reporting to stakeholders and users 
of assistance. Part 3 focuses on lessons learned and the outcome achievements 
of the monitoring and evaluation effort. Among the lessons learned were that 
an enabling environment for monitoring and evaluation requires senior 
management commitment and willingness to learn and that performance measures 
need to be defined for both service and program delivery. The need for the 
involvement of program customers from the beginning and the provision of 
adequate resources for monitoring and involvement are additional lessons 
that, like the other conclusions from this project, have implications for 
educational evaluation and the development and use of performance measures. 
Three appendixes describe SIGMA's development, methodological concerns, and 
sample methodology. (Contains 2 tables, 1 figure, and 14 references.) (SLD) 
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Synopsis 



This paper focuses on the experience of the Policy Education Centre on Assistance to 
Transition (PECAT) in providing monitoring and evaluation (M&E) services to SIGMA 1 , a 
joint OECD/PHARE 2 program. Part 1 of the paper describes the challenges of the 
experience in terms of context, program design and performance monitoring. In a very 
dynamic and changing environment, clients followed individual and very differing reform 
paths, requiring assistance to be highly tailored and constantly adapted to new realities in 
policy, programs and staff. The SIGMA program was intentionally non-prescriptive, offering 
clients a menu of options regarding the types, velocities and sequencing of reforms. As the 
goals of the assistance program were not clearly defined, the evaluators were requested 
to assist SIGMA in developing suitable performance measurements. 

Part 2 of the paper provides a description of how certain problems were addressed, such 
as the design and implementation of the M&E strategy, the process of establishing 
performance measurements, and the development of tools for data collection. The 
challenge to the evaluators was to build suitable performance standards for reporting to 
stakeholders and users of assistance (representing different levels of government and 
having different reform needs). 

Finally, part 3 focuses on lessons learned and outcome achievements of the M&E effort, 
SIGMA’s hightened awareness and commitment to M&E and their internalization of new 
formats to permit self-monitoring and assessment. At the operational level, this meant the 
successful use of information collection methods to keep up with the rapidly changing 
environment, the development of service performance measures acceptable to SIGMA, 
and SIGMA’s stakeholders increased participation in program policy orientation. 

Other lessons learned include the recognition that an enabling environment for M&E 
requires senior management commitment and willingness to learn and that performance 
measures need to be defined for both service and program delivery. The need from the 
beginning for significant involvement of program customers in M&E and provision of 
adequate resources for M&E services and functions also emerged as important lessons. 



Support for Improvement in Governance and Management in Central and Eastern European 
Countries 

The Organization for Economic Cooperation and Development based in Paris. The PHARE 
(Poland and Hungary: Assistance in Restructuring Economies) program was originally founded 
to assist Poland and Hungary in the restructuring of their economies but has since become the 
main European Union’s mechanism for providing assistance to the Central and Eastern 
European countries. 



Introducing Performance Measurements 
in the Evaluation of Assistance to 
Public Administration Reform 
in Central and Eastern Europe 



I. Statement of the Challenge 

With the end of the Cold War, many ex-communist countries began the transition to new 
economic, social and political systems and realities. In response there was a strong 
promise of Western support to assist these countries in reaching their goals. What began 
was one of the most unprecedented periods of change in peace time in the Region, 
characterized by great expectations, little if any relevant experience, significant differences 
of views on what and how things should change and extensive periods of administrative 
and political turnover. 

Countries in transition and donors faced extraordinary challenges: howto determine needs 
and prioritize them, how to transfer concepts new for these countries into practice, how to 
develop new skills and ensure a culture shift and how to adapt to a constantly moving 
reality. This paper focuses on one experience in monitoring and evaluation (M&E) in 
Central and Eastern Europe (CEE) in which evaluators and program implementors had to 
adapt to and cope with the challenge of this context, the program design and a lack of 
suitable performance measures. This part of the paper elaborates on some of these 
challenges. 

A. Context and Customers 

Regional Scope 

The years of transition since 1989 have demonstrated the sheer magnitude and complexity 
of the conditions under which CEE countries 3 are striving to transform their public sector 
systems, economies, and societies. Each of the CEE countries has reached a varying 
stage of transition. They were affected by communism in different ways and their transition 
trajectories, political choices and economic strategies reflect their separate transition take- 
off points. These new democracies and market economies continue to face a very difficult 
external economic environment and unprecedented internal transformations, especially as 
related to the modernization of the public sector. 

When assistance began to the region, it was often based on Western “generic” notions of 
what was needed. Neither recipients nor donors understood in any degree of detail what 



Generally includes the three Baltic Republics, Poland, Hungary, Czech Republic, Slovak 
Republic, Bulgaria, Romania, Albania and Slovenia. 
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public administration reforms should entail. In the early years (1990-1993) movement in 
this sector was slow. Developments in reform, the internal dynamics of transition and the 
external environment eventually brought a confluence of agreement and an agenda for 
public administration reform (PAR), especially in countries hoping to be first integrated with 
Western Europe. ■' ; 

Public Administration Reform 

The experience of the OECD countries shows that public administration reform is a long- 
term, multi-dimensional process, particularly when it concerns the overhaul of entire 
systems and structures of government. Given the mosaic of possible reform directions, 
there is a need to search for good experiences and options for building good governance. 

As each CEE country entered transition the predominant view was to significantly reduce 
the role of the State. This has been recently replaced by a more mature understanding of 
the role of good governance for an efficient and competitive market economy. Political 
elites have now begun to recognize the need for a modern public administration. 

There now exists in the majority of CEECs more political stability, which provides more 
opportunities for carrying out public administrative reforms. 4 Consequently, the period of 
crisis management is receding and governments can now focus on strategic management 
reforms and their implementation. At the same time a high turn-over in public 
administration staff, caused by emerging opportunities in the new market economies, make 
the reform process difficult to achieve. 

The last years of transition have highlighted the need for reform of the State. The 
complexity of the social/financial difficulties requires that the current, fragmented approach 
to reform should be replaced by a more comprehensive modernization of the State. This 
is true especially for the countries which in varying degrees have been exposed to a crisis 
in public finance. 

Internal Dynamics 

Since 1 989, Central and Eastern European governments have concentrated on dismantling 
the inherited communist political and administrative structures and their command 
economies, rewriting laws to facilitate the emergence of democracy and free markets, and 



Particularly during the first years of transition in many countries, with the exception of the Czech 
Republic, political instability was observed in the Region, which caused frequent modifications of 
personnel and policy and presents reformers and providers of assistance with a constantly 
shifting constituency and, consequently, with a continually fluctuating demand for services. 
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responding to the resulting shocks that have emerged. The changes have proven more 
difficult and more diverse than were originally anticipated. 

The first phase of transition, marked by impressive gains in macro-economic stabilization 
and liberalization, was largely led by central governments .with the support of Western 
assistance. These top-down, government-led transformations have now lost momentum 
by the end of their first phase. In some countries there has been limited progress with the 
reform of PA procedures and laws. This situation may reflect the fact the macro-economic 
reform packages were not accompanied by comprehensive institutional reform packages. 

By now countries such as the Czech Republic, Hungary, Poland, Slovenia and to a lesser 
extent the Baltic States have advanced their reforms quickly and deeply to meet EU 
accession requirements. Others like Bulgaria and Romania are making fresh starts to 
revise their transition strategies; still others like Albania and the Slovak Republic have 
moved away from their initial transition paths. A slower pace of reform has emerged, even 
among advanced countries in certain sectors. If the slow-down of reform cannot be 
overcome, a danger exists that the CEE countries will stagnate at different stages in their 
transition, retaining many structural features from their communist past. 

External Dynamics 

With transition the CEE countries are becoming increasingly involved in international 
structures. Opportunities opened for membership in GATT, the World Bank, the Bank for 
International Settlement (BIS), the International Monetary Fund (IMF), and participation in 
structures of the Organization for Economic Cooperation and Development (OECD), such 
as the Centre for Cooperation with Economies in Transition (CCET) and association 
agreements with the European Union (EU). Appropriate country policies have had to be 
formulated to deal with a little known external environment. Interchange with the Western 
external environment has meant learning to define needs, secure support, negotiate and 
participate in ways unfamiliar to CEECs and for which they were, to a large extent, 
unprepared. 

For the countries in the region, integration with the EU constitutes a paramount priority 
which has broad public and political party support. Even parties built from the previous 
communist ones have adopted the integration agenda. In signing the Europe Agreements 
with many CEE countries the EU has, in principle, acknowledged these states’ intrinsic 
right to full membership in the Union. The CEE countries have increasing access to the EU 
market and are transforming their political, economic and social systems so as to meet EU 
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entry criteria. Supporting this is EU technical and other assistance, notably under the 
PHARE 5 program. 

"Dreams of European Union" are an important psychological factor in the domestic debate 
on economic and political developments. These aspirations provide a positive political 
environment for market-friendly economic policies, human rights policies and 
democratically-oriented solutions. 

The major goal of EU assistance to the associated countries is progressive preparation for 
“integration into the internal market of the EU, through phased adoption of the Union’s 
internal market acquis". 6 The PHARE program assists the pre-accession measures: the 
development of infrastructure and the promotion of intra-regional and environmental 
cooperation. PHARE helps the CEE countries absorb the “ acquis communautaire ", and 
complete market and public sector reforms so as to create the conditions, as laid down in 
the EU White Paper, required for future membership. 7 

The OECD is increasingly outward-looking. Since first directing its attention to the region, 
it has extended its focus to assist all the CEE post-communist countries by providing 
support to developing a policy dialogue with these countries and by broadening 
intergovernmental cooperation to cover most economic, social policy and public 
administration domains. This program of support combines OECD and EU/PHARE 
assistance and is described in the next section. 

B. The SIGMA Program 

In May 1992, the SIGMA program was established as a joint initiative of the OECD/CCET 
and EU/PHARE to assist six Central and East European countries in the design and' 
implementation of reform efforts for public administration. A major reason for launching 
the initiative was the assumption of being able to replicate the experience the OECD Public 
Management Service (PUMA) had gained with European Commission countries, such as 
Greece, Portugal, Spain and Turkey, in improving their systems of public service. The 
SIGMA program was expected to contribute to the CEECs transition to democratic and 
effective governance. 



The PHARE (Poland and Hungary: Assistance in Restructuring Economies) program was 
originally founded to assist Poland and Hungary in the restructuring of their economies but has 
since become the main EU donor mechanism for providing assistance to the CEECs. 

See European Council, Meeting on 9-10 December 1994 in Essen, Presidency Conclusions, 
Brussels, December, 1994, p. 8. 

Ibid, p. 23. 
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SIGMA 1 (1992-1994) was evaluated in September 1993 and considered successful. 
Consequently, it was decided to continue the program and to expand it to five more 
countries. A major recommendation of the evaluation was the establishment of 
independent monitoring and evaluation services. Following the sixth meeting of SIGMA's 
Steering Committee (also known as Liaison Group), 8 PECAT was selected by an open 
tender to provide such services for SIGMA 2 (1994 - 1996). 

Table 1 summarizes the components and evolution of SIGMA's program design in Logical 
Framework form. It was compiled from the Liaison Group’s program orientations, SIGMA 
1 evaluation recommendations and related policy documents. Significant phase two 
changes in design included: the addition to the mandate of "support development of 

efficient and effective public institutions", the concept of "in search of good governance", 
increase in country coverage to a total of eleven 9 , re-prioritizing sectoral services and 
decreasing regional activities in favor of more individualized country projects. Other 
differences related to better tailoring services to client needs and changes in inputs in 
terms of organization and input performance measures. 

SIGMA's service standards were (and continue to be in phase three): timeliness, rapid 
responsiveness, flexibility and high technical quality delivery, which are achieved through 
a core resource staff and maintaining networks of experienced professionals. Program 
services focus on the core management systems of government, namely: country reform 
strategy (sector 1), management of policy making (sector 2), expenditure management 
(sector 3), public service (sector 4), administrative oversight (sector 5) and information 
services (sector 6). Under SIGMA 2, the Liaison Group re-prioritized the service emphasis 
on sectors 2, 3, 4 and 6, as a result of resources constraints. Services are provided 
through traditional forms of technical assistance in multiple and continuous interventions: 
technical advice, training opportunities and, to a very limited extent, equipment; 
Considerable expansion in information sharing has allowed SIGMA to produce specialized 
products in the priority sectors and publish monthly progress reports and a quarterly 
newsletter. 

The institutional and financial arrangements constitute a significant feature of the SIGMA 
technical assistance program. Built on the PUMA service concept, experience and network 



The composition of the Group were representatives from each CEEC, donors and PHARE. 

Under SIGMA 2, the Group meets once a year, the agenda is prepared by SIGMA in 
consultation with the Group and the meetings are chaired by the head of PUMA. 

SIGMA 1 countries: Bulgaria, Czech Republic, Hungary, Poland, Romania, Slovak Republic; 
SIGMA 2 countries: the SIGMA 1 countries plus Albania, Estonia, Latvia, Lithuania and Slovenia. 
On January 1, 1993, Czechoslovakia split into the Czech and Slovak Republics. 
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of experts, SIGMA offered a menu of short-term (a few days to a week), strategically 
focused assistance in response to specific client needs. SIGMA followed standards of 
excellence and was strongly customer oriented. It was assumed recipients benefiting from 
such interventions could advance reforms on their own while being guided at critical stages 
by SIGMA expertise. v 

SIGMA funding arrangements have been unique in the OECD and heavily dependent on 
external financing, especially EU/PHARE: 82% of total support for phase 1 and 86% for 
phase 2. The PHARE backing assumed that SIGMA would become a lead technical 
assistance developer, providing the initial identification and planning support to customers. 
Once project concepts had matured sufficiently, they would be “passed on" to PHARE or 
other donors for major funding. 

The difference in assistance philosophy between SIGMA and PHARE led to difficulties in 
communication, program implementation, coordination at country level and periodic 
resource mobilization. SIGMA had to carefully negotiate funding for the program while 
maintaining credibility and confidentiality with potential customers (e.g. cabinet offices, 
finance ministries, public administration offices). 

Stakeholders, particularly PHARE, demanded that SIGMA show program results and relate 
these to other donor support efforts. The challenge for SIGMA lay in preserving program 
components which favored customers and its own way of doing business while 
demonstrating the program had achievements, was true to its mandate and delivered 
services appreciated by its customers. The evaluators’ challenge was how to merge 
SIGMA’s strengths into a management process which focused on showing performance. 

C. Measures 

SIGMA attempted during phase 1 to carefully document project formulation and 
implementation. It soon became apparent the multiple, small, short-term interventions did 
not lend themselves to the classic notion of "projects". SIGMA staff spent inordinate time 
and effort developing the documents and even forwarding them to PHARE for information 
purposes. It became clear that this approach simply did not contribute to performance 
management or effective reporting to stakeholders. The problem was two-fold: the units 
of action were too small to merit the level of effort and the "projects" generally lacked 
performance measures to provide SIGMA and stakeholders adequate feedback on 
problems and progress. It was also difficult to determine what constituted poor or 
successful performance. As a result, SIGMA 1 monthly reports provided information on 
"intervention events" by country and sector without giving a sense of the kind of 
contribution and/or continuity of purpose of such services. 
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At the overall program level, SIGMA 1 also had difficulties in showing performance as the 
design had very little in the way of measures with a related baseline which could be 
monitored and evaluated. This was less problematic at the input level, but was evident at 
the output and especially at the outcome level. 

By SIGMA 2, these shortcomings still had not been resolved. Although the mandate was 
made somewhat more focused by the addition of “good governance” and more guidelines 
for program priorities, the design lacked specificity in program expectations, especially in 
terms of outcomes. The evaluators needed to develop performance measures, sufficient 
to maintain program flexibility and responsiveness, and to do this in partnership with 
SIGMA. 

The M&E service to the program had to take into account not only the rapidly changing 
transition context of each country in the Region. It also had to incorporate explicit 
recognition that small, “low-risk” interventions in support of public administration reform and 
“good governance” in particular, were small steps contributing over a varying time 
continuum to reforms and capacity building rather than a specific product-oriented process. 
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II. PECAT Approach/Strategy 

The SIGMA 2 program was underway six months when PECAT began to provide 18 
months of M&E services. This meant having to “catch-up” with established management 
systems and identify opportunities where the M&E contribution could be the most effective. 
The work was divided into four phases: 1) setup, management information and institutional 
arrangements: 2) regular monitoring: 3) interim evaluations and preliminary findings; and 
4) final program evaluation. This part of the paper describes the adopted methodology, 
data collection and analysis. 

A. Methodology 

It has been said, evaluation is an art and consists of selecting the right mix of methods, 
procedures and measurements. In the SIGMA monitoring and evaluation work, this meant 
formulating approaches and M&E methods which would meet the M&E objectives, resolve 
certain methodological constraints and enable both management and evaluators, as 
partners, to improve their respective functions. 

The objectives of PECAT’s work were to: 

a. improve the effectiveness and efficiency of SIGMA in implementing the 
assistance program by providing SIGMA’s management independent 
monitoring and evaluation of program performance: and 

b. develop program M&E tools worthy of replication by exploring the application 
of performance measurement and participatory evaluation. 

To address these objectives, the evaluation strategy consisted of four different 
perspectives: 

i) assessment of SIGMA program design; 

ii) assessment of performance in terms of meeting the programming and 
management directives resulting from Liaison Group decisions; 

iii) assessment of performance, on a program-wide level, in terms of standard 
evaluation criteria (efficiency, effectiveness, significance); 10 and 



These terms were defined as follows: 

efficiency: whether the outputs justify the cost, whether there are alternative means of 

achieving similar outputs; 

effectiveness: how well are benchmarks and outputs being met in a constantly changing 

environment, what is the quality of service responsiveness to country needs, 
what are the causes for success or failure; 

significance: will program achievements contribute to reform or other higher (broader) goals 
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iv) assessment of performance in terms of six specific service performance 

measurements at the output level. 

The measures which SIGMA agreed upon were: 

• Follow-up/Phasing: The extent to which SIGMA, intervention within one 
project is related to and build on earlier interventions within a reasonable, 
appropriate time frame. 

• Penetration: The extent to which the benefits of SIGMA actions are spread, 
horizontally and/or vertically, beyond the immediate client or participation 
level. 

• Cooperation with other donors : The extent to which SIGMA reaches out to 
other donors to achieve complementarity, to participate in, to build upon, or 
in some way connect to a SIGMA intervention. 

• Ownership: The extent to which the actions are defined and driven by the 
client for whom SIGMA provides services. 

• Cost-efficiency: Relates to the comparison of the relative magnitude of costs 
among sample projects, and with the apparent value of outputs (or likely 
benefits) of similar initiatives supported by other donors (while recognizing 
the typology of the SIGMA organization). 

• Contribution to the continuation of the reform process: The extent to which 
SIGMA interventions promote and keep active public administration reform 
(Keeping the "ball" active). 

A series of questions were formulated which stakeholders and SIGMA wanted answered 
for different monitoring, interim assessment and final evaluation reports. These provided 
the evaluators guidelines on what data was needed and when. 

The evaluators addressed certain methodological concerns which could contribute to the 
introduction in SIGMA of objective standards for monitoring and evaluation. Among these 
were the treatment of baseline data, use of program performance measurements, and 
assessment of partial versus total program coverage. Annex 2 describes how these issues 
were treated. 

Finally, a major methodological aspect which permeated the M&E work concerned 
developing a partnership with SIGMA. Every aspect of M&E design and major 
implementation tasks were jointly reviewed. SIGMA provided invaluable support and insight 
with regard to data availability and collection. PECAT demonstrated the importance of 
more rigor in specifying project expectations in measurable terms. Points of difference such 



beyond the program objective, what are the strengths and weaknesses of the 
program over possible alternatives, what side effects may be occurring. 
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as use of sector and country PAR measurements were generally resolved. PECAT 
products were timed to correspond to SIGMA and PHARE decision-making periods and 
events. 

B. Data • 

The M&E work was organized to collect data from four sources: I) secondary data from key 
and selective SIGMA documentation and primary data from interviews with SIGMA staff; 
ii) secondary data (all available) from documentation, specifically related to the sample and 
primary data from interviews with SIGMA staff and others (clients and donors involved in 
the fifteen sample projects); iii) primary data from questionnaires directed to persons who 
participated in SIGMA sponsored conferences and workshops; and iv) primary data from 
21 1 interviews with both clients and other donors to elicit ratings of SIGMA products and 
performance. 

Review of SIGMA documentation, the first source of data, highlighted strengths and 
weaknesses in performance management. This work provided a close view of the diversity 
of client requests: e.g. a technical query requiring on the spot response, research of the 
most suitable approach in organizing a cabinet office, options to procure expertise in 
designing a procurement system, developing capacity to prepare for EU accession, etc. 
This experience led to selecting a representative sample of SIGMA services on which data 
could be collected at project level and subsequently related to program impact. Data 
collected from participants of SIGMA sponsored conferences and workshops indicated 
client satisfaction also at project level. This data was then combined with data from 
interviews from a wider audience. 

SIGMA and the evaluators encountered difficulties at times during the data collection/ 
Process. However, it also contributed to a better understanding of respective needs and 
capacities during the M&E partnership. 

C. Analysis 

Bearing in mind the above considerations, the evaluators adopted the following 
combination of analytical techniques: 

1) project design analysis 

2) strengths, weaknesses, opportunities and threats (SWOT) analysis 

3) a statistically random sample of projects 

4) judgmental assessment 
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Use of Project Design 

The SIGMA program consists of a set of resources and activities directed toward objectives 
articulated in the SIGMA Mission Statement and the Conventions signed with program 
supporters, especially EU/PHARE, other relevant SIGMA program design documents 
approved by the SIGMA Liaison Group and the OECD/CCET. The analytical activities 
involved monitoring and evaluating the application, utilization, response of SIGMA and its 
resources in relation to the SIGMA program design. Initially, the task required the review 
of all relevant documentation to “assemble" the project design. Subsequently, an 
assessment of performance was made of the use of “inputs” (financial, human and 
material) and the production of “outputs” (results generated directly from the utilization of 
inputs i.e. provision of services). 

For the bi-monthly monitoring reports this analytical technique provided useful feedback 
to SIGMA to permit follow-up and corrective action. The major data sources consulted for 
this purpose were SIGMA Monthly Activity Reports and persons interviewed - when 
possible. For the two interim assessments (October 1995 and February 1996) and the final 
evaluation report (July 1996) the analysis included examination of performance in relation 
to efficiency, effectiveness and significance. It combined data on inputs and outputs as well 
as customer satisfaction at project and program outcome levels. 

Use of SWOT 

To apply this technique it was necessary to identify and develop variables reflecting SIGMA 
service related performance. The analysis consisted of determining SIGMA's SWOT profile 
using these SIGMA-PECAT agreed upon variables. The technique proved particularly 
useful for the case studies as well as for examining program impact. 

Use of Project Sampling 

During the period of review, SIGMA was involved in approximately 135 projects. Instead 
of trying to examine all projects, it was decided to select a mathematically random sample 
to assess program performance at project level and from these results determine the 
implications on program impact. To draw such a sample of projects, the following M&E 
considerations were taken into account: 

• M&E must be cost-effective; 

• the risk of overall evaluation errors must be very limited (1%); 

• the evaluation risk is a product of inherent risk, internal organization and sampling 
risk; 

• mathematical sampling is a generally accepted evaluation technique; and 
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• the ongoing nature of monitoring and evaluations allows for adjustments and 
extended assessments in a following period. 

Two samples at different intervals (August and December 1995) were chosen. The 
sample size was calculated on the basis of the principle that the reliability of this 
assessment should be at least 80% (i.e. 20% sample risk). Furthermore, to emphasize the 
requirements for the cost/benefit analysis of SIGMA projects, a monetary unit sampling 
method was used so that more costly projects would have a higher probability to be 
selected as part of the sample. The sample size from the combined two selections 
amounted to 15 projects, representing 23.3% of the total direct program costs. The sample 
also had good representation by country and by sector. Annex 3 contains more detailed 
information on the sampling process. 

To complement the sample data, the major forms of SIGMA 2 project interventions were 
also mapped to determine overall program service intensity by sector, spread of projects 
and trends. This provided a profile of the type of services SIGMA customers requested. 

Use of Judgmental Assessment 

As project data in the sample sometimes had omissions, the evaluators thought it 
necessary to complement the analysis of project performance with judgmental assessment. 
This approach does not involve collecting a great deal of new data or analyzing masses 
of existing data, techniques which are expensive and very time-consuming. It is an 
approach which estimates program impact and is particularly suited to programs when The 
judgmental assessment was carried out by combining different data sources, including the 
satisfaction rates, limited M&E funds are available, when no pre-intervention measures 
exist, so that reflective controls cannot be used and when neither randomized nor 
constructed controls can be used. In this approach, the judgements of presumed experts, 
program administrators or participants play the major roles in providing estimates of net 
impact. 11 

The judgmental assessment was carried out by combining different data sources, including 
the satisfaction rates. These “information inputs” were then analyzed in relation to each 
other to identify patterns of similarity in judgement and triangulation. These results were 
then “weighed” by the evaluators to provide an independent assessment. 



See Peter H. Rossi and Howard E. Freeman, Evaluation a Systematic Approach, "Strategies for 
Impact Assessment”, Newbury Oark, CA:1993, pp. 215-259 
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III. M&E Outcomes and Lessons Learned 

As noted at the beginning of this paper, the provision of assistance under the SIGMA 
program as well as monitoring and evaluation services constituted an unique experience 
for customers, program staff and advisers, stakeholders and evaluators. The contextual 
dynamics and near “virgin” territory for recipients and donors to embark on paths of 
change, often poorly defined, provided an exciting challenge. The “inputs" provided to 
public sector reform take time to produce and manifest the expected outputs and results. 
Hence, outcomes resulting from these partnerships will be found, in many respects, still in 
the future. 

This part of the paper summarizes the main known impact of the SIGMA program, 
describes the involvement of SIGMA and PECAT as partners and reviews the relations of 
stakeholders and customers in the M&E function of the program. The chapter supports the 
research of others that “the form of, and resistance to, effective performance monitoring 
systems is contingent on the relevant organizational considerations and the related 
environment.” 12 

A. Achievements 

Bearing in mind the changing face of transition in the CEE region and individual countries, 
SIGMA 2 continued to provide services assessed successful in terms of efficiency, 
effectiveness, significance and program impact. Project design expectations, despite being 
vague at times, were attained. The pattern of services rendered, by sector and type of 
activity, did not substantially change over time. All eleven countries were covered by 
SIGMA interventions, the main kind being missions (travel to provide advice), study tours 
and workshops, seminars and/or conferences. Table 2 summarizes such assistance by' 
country and month. 

Efficiency : A comparison of the relative efficiency of different interventions shows that 
SIGMA service costs can be considered reasonable compared to similar efforts supported 
by other donors. Multi-country activities (generally training efforts for more than one country 
recipient) rank expensive with the meetings of the Liaison Group being the most costly. 
SIGMA has made an effort to diversify its sources of program funding, which improved 
overall efficiency in resource utilization. However, the overall dependency on PHARE has 
not diminished nor has SIGMA shown a high sensitivity to cost considerations when using 
“expensive" experienced Western practitioners. More attention to finding specialists in the 



John Mayne and Eduardo Zapico-Goni, eds. 1997. Monitoring Performance in the Public Sector, 
New Brunswick, USA: Transaction Publishers, p. 19. 
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region and involving clients in preparatory project planning were areas of weakness in 
keeping costs down. 

Effectiveness: Although there were limited weaknesses in efficiency, SIGMA 2 improved 
consistently, during the eighteen months of M&E services; the positive client satisfaction 
identified in phase 1. Over 150 interviews among clients and donors were carried out and 
resulted in “satisfactory” to “very satisfactory” ratings. SIGMA direct clients and 
counterparts showed continued and growing appreciation of the program’s assistance 
while donors lagged slightly behind in these ratings, as can be noted in Figure 1. 



FIGURE Satisfaction Rate 




Participants who attended SIGMA sponsored seminars, workshops and conferences rated 
the quality of service inputs ( expert presentations, documents provided) as high in 70% 
of cases, with the rest being judged as medium. For projects implemented in only a single 
country, the satisfaction rates for the quality of organization and documents provided by 
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SIGMA were "high” in 83% of cases while expert presentation received unanimous high 
ratings. Hence the high appreciation of the program is evident, even when considering 
some participants made modest financial gains and donors had at times insufficient data 
to rate SIGMA. 

Among the “input” factors contributing to the high rates of satisfaction were: SIGMA’s high 
quality of in-house expertise, use of the PUMA connection and network of external experts 
and practitioners (e.g. peer reviews of draft laws, sharing experiences on EU accession), 
turning to external experts from the region for services to countries further behind in their 
transition achievements (10% of total consultancy days, the balance being from OECD 
countries), and seconding young CEEC professionals to SIGMA for short- to medium-term 
assignments. The most recognized “outputs" by SIGMA clients included multi-country 
events; opportunities to develop networks of practitioners and experts within CEE; SIGMA 
publications and wide dissemination of monthly reports, a quarterly newsletter ( Public 
Management Forum), public management profiles, and other technical and marketing 
products; and a “reference library” of laws and regulations (for comparative purposes) and 
over 2,000 items of particular interest to CEECs. The multi country events especially 
provided opportunities for CEE participants to learn from each other about the potential of 
specific reforms and how further to use SIGMA service and to develop their respective 
approaches to PAR. These events also proved successful in initiating interest in PAR, an 
important development for when this had low priority. To promote complementarity in 
program operations, SIGMA contributed successfully to better strategic use of the available 
mix of assistance to PAR in the CEECs. The program also helped accelerate preparatory 
work required to initiate national assistance programs to PAR by other donors. 

SIGMA also improved its own image, becoming better known in the region among clients 
and donors, especially PHARE. The monthly reports provided prospective information, an 
indicator of more internal planning than mere response to requests. 

Besides these effective program and management results, SIGMA fell short in meeting 
certain expectations. For example, coordination of PAR initiatives among donors could 
have been stronger and prevalent in more countries. While there were examples in which 
SIGMA was known to have been the main coordinator, at PHARE the opinion continued 
that the program attained less than was desired. Another weakness related to being able 
to demonstrate more clearly how SIGMA outputs resulted in specific PAR initiatives. The 
program continued to resist developing PAR progress performance reports by country and 
sector (considered too sensitive a matter). Hence, SIGMA effectiveness could be 
measured more in the quality of service delivery, which was highly valued, but less in 
substantive program outputs and could not be easily aligned to context and PAR 
movement. 
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Significance: SIGMA has encouraged, nurtured and assisted the PAR process in the 
CEECs and its clients considered the program made a “significant difference” in the pursuit 
of its mandate of “good governance”. Operating often jn an unstable, politicized 
environment, the value of SIGMA responsiveness to client needs has been sometimes 
offset by the tendency to optimize windows of opportunity rather than pursue more actively 
PA reforms or coordination among donors in the area of PAR. As a result, SIGMA has not 
yet been able to achieve the expected catalytic role by contributing to limit the 
fragmentation, on both the recipient and donor sides, of PAR initiatives. 

Despite the lack of country-specific PAR program performance indicators, already 
mentioned, which made it difficult to assess the “value-added ness” of SIGMA in reference 
to PAR movement, progress in achieving certain PA reforms and/or sustaining the 
dynamics of reform and agenda has been attributed to SIGMA service provision. SIGMA 
raised the priority and visibility of PAR in the CEECs by promoting the establishment of 
reform cells in the relevant country central government administrations. As a result, many 
countries formulated national PAR strategies with SIGMA assistance, which otherwise 
would have occurred later or not at all. The following reviews by sector more specific 
results attributable to SIGMA. 

/ 

Among the most wide-reaching efforts, in the area of Country Reform Strategy (Sector 1), 
has been SIGMA’s support to the Polish Government Plenipotentiary for State Economic 
Administration Reform where a comprehensive structure was elaborated and was adopted 
by the Sejm in June 1996. Similar, though perhaps less comprehensive, was SIGMA 
support to organize the functions and operations of the Latvian Government chancellery 
and the Albanian Prime Minister’s office. These SIGMA contributions were identified by 
government officials as being of critical importance. 

Contributions to the Management of Policy Making (Sector 2) are illustrated by SIGMA’s 
pioneering work on internationalization of policy making with particular emphasis on the 
coordination of policies related to European accession. Significant SIGMA interventions 
were a series of seminars, conferences and experience sharing events which focused on 
different aspects of the political and administrative process 13 and the preparations the 
CEECs would have to begin to undertake. 

In Expenditure Management (Sector 3) SIGMA is recognized to have brought about CEEC 
awareness (together with the International Monetary Fund and the World Bank) to: 1) 
establish the integrity of the national budgets; and 2) capture control of all public money 



For example, multi-country workshop on the Internationalization of Policy-making, a workshop 
on CEE governments’ capacities to coordinated domestic and international (particularly with 
regard to EU accession) aspects of policy-making. 
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in a single account with a centralized payment system under the control of a Treasury as 
well as the implications of such a move for further reform needs. 

Currently the documents on PAR, drafted in CEECs more frequently include reference to 
and/or emphasis on the relationship to budgetary and financial management reform. 
SIGMA was particularly active in promoting this approach and emphasizing such a linkage. 

During the early period of SIGMA 2, assistance to the development of national 
procurement systems (based on international standards) provided a foundation for PAR 
in this sector. SIGMA successfully organized interventions through which Western as well 
as CEE (Polish) experiences could be shared among CEECs. The multi-country work 
served to place public procurement “on the agenda" and to raise awareness of a sequence 
of related policy and implementation issues. 

The reform of Civil Service (Sector 4) was considered, since the inception of SIGMA, as 
among the more critical concerns to the overall PAR process. Assistance focused in the 
beginning on training systems for top civil servants, then it moved on to the establishment 
of civil service laws (including reform of pension, career development and remuneration 
systems), and most recently to staff reduction and administrative modernization. As other 
donors, particularly PHARE, are active in the area of training in public administration, 
SIGMA focused on optimizing strategies and the content of training as a means for 
improving performance capacities in public services. 

With regard to SIGMA’s Sector 5, recent increasing client priority attention concerning 
administrative control and administrative procedures and oversight institutions led to the 
preparation by SIGMA experts of technical papers on systems in these areas in EU and 
five selected CEE countries (Albania, Bulgaria, Estonia, Hungary and Poland). This work 
provides the CEECs and interested practitioners with reference material and a certain 
PAR-baseline which hitherto was unavailable. 

The relevance of the objective of SIGMA is supported by two important developments. 
First, the simplistic anti-state mentality in CEE countries in the early days of transition has 
gradually been replaced by a more mature understanding of the role of good governance 
for democracy and an efficient and competitive market-economy. Second, the European 
accession has already become a drive which propels countries in transition to prepare their 
respective pre-accession strategies which both include and depend on PAR. As a result, 
the second phase of transition holds greater chances for actions aimed at good 
governance. Given the fact that SIGMA is one of the few assistance programs in the 
region specialized in providing support to the reform of core management systems, the 
objective of SIGMA continues to be relevant. 
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From the perspective of performance monitoring, the achievements and limited 
shortcomings of the SIGMA program show that a public organization with high uncertainty 
(in context, customer demand, funding) and complexity of tasks can benefit from the 
gradual introduction of performance measurement. The experience would suggest the 
need for continual refinement and updating to maintain their significance. 14 

B. Involvement of SIGMA 

While SIGMA 1 followed monitoring processes, producing periodic reports and even 
attempting to carry out project design principles, the program had not developed a sense 
of performance management in the most recent sense of the meaning. The fluid context 
in the early years combined with the realities of mounting a new service to a clientele 
largely unknown by the West established patterns of operations which gave low priority to 
documenting service delivery in a performance measured way. The conditions under 
which SIGMA developed were marked by confounding factors which made defining, 
monitoring and use of performance measurement at times difficult: uncertainty, diversity, 
interdependence and instability. 15 

These factors gave rise to difficulties in designing and implementing performance 
monitoring for SIGMA 2. Uncertainty played an important part in terms of determining the 
relationship between SIGMA outputs and outcomes, especially with regard to PAR. The 
external changing environment, over which SIGMA had no control, evoked preferences by 
management to keep the performance monitoring focus on outputs rather than attempt to 
envision possible reform outcomes which had high risks of ever happening. Diversity was 
prominent in the number of multiple customers and their divergent interests in each of the 
eleven countries but also among them and over time. SIGMA often was caught in the 
middle when there was no consensus on goals and values, leading to the provision of 
services which focused on very small rather than broad reform actions. This made defining 
success a problem and obtaining agreement on program performance indicators illusive. 
It has been observed that such lack of consensus on policy objectives of reform or even 
conflicting basic values might explain why more progress often occurs in defining 
measures at service level (Mayne and Zapico-Goni, eds. 1997). 

Factors relating to interdependence meant taking into consideration the actions of SIGMA 
clients as well as other donors. Hence, one important indicator for service delivery was the 



N. Carter, 1991. “Learning to Measure Performance: The Use of Indicators in Organizations,” 
Public Administration 69: pp. 85-101 . 

These factors were identified through a literature review and discussed in Mayne and Zapico- 
Goni, eds. (1997). 
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extent to which projects reflected coordination with other donors and/or customers. This 
became particularly difficult in countries or institutions with high instability for political, 
financial or other reform reasons. The M&E work demonstrates that the four confounding 
factors explain in large part resistance to and difficulty in defining and implementing 
performance monitoring — a conclusion similarly reached by Mayne etal. 

The evaluators sought to involve SIGMA management and staff to adopt a continuous and 
interdependent learning approach to managing their operations and incorporating 
performance information generated by the M&E work, seeking whenever possible to 
overcome such confounding factors. 

A detailed work plan was prepared by the evaluators and approved by SIGMA which 
included as a first joint activity a workshop. It was held to convey M&E concepts and 
methodologies to program staff and management and to begin developing the partnership 
for building performance monitoring and ultimately performance management. The 
objective was to ensure both partners would have a common understanding and 
involvement from their respective roles with regard to what would be measured, what data 
would be needed, how the analysis would be conducted, and what would be reported. It 
was decided the bi-monthly monitoring products would provide SIGMA performance 
information suitable to daily operations while the interim and final reports would assess 
periodically performance related to outputs and impact. SIGMA in turn would provide not 
only necessary data but also alert the evaluators to special requests from clients which 
would benefit from the presence of an independent observer. The involvement of SIGMA 
in defining this partnership helped build trust and a sense of what information needed to 
be shared. Eventually, a common understanding evolved, on these matters which allowed 
both sides to carry out their respective functions without having to explain the reasons of 
why the data was necessary or how it would be used. 

Another major objective of the workshop was to attempt to get SIGMA to define service 
and program performance measures as opposed to having the evaluators establish ones, 
which later would have little ownership and most important utilization in program decision- 
making and service delivery. Initially, these efforts were only marginally successful. Staff 
and management acknowledged the need and potential use of performance monitoring; 
however, they had great difficulties in defining measures relevant to the services they 
rendered and program delivery, in general. Distances between the location of SIGMA 
(Paris) and the evaluators (Warsaw) also posed constraints on follow-up and continuing 
the dialogue. The suggestions which emerged became the nucleus upon which it was 
possible to build and refine the performance measures and particularly use them in the first 
interim-assessment. 
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Subsequently, the evaluators conducted their first series of on-site data gathering in the 
eleven program countries. SIGMA also encouraged the evaluators to schedule their visits 
to countries in such a way as to allow evaluators to attend program single- or multi-country 
events. This gave opportunity to observe SIGMA in action and to interview clients on the 
spot. The visits were followed up with a rapid appraisal of participants in similar events, 
the review of case studies and the consolidation of analytical work. 

After a draft interim report was shared with SIGMA, it became evident to management and 
staff how the results can be used for improving operations and reporting to SIGMA 
stakeholders. The evaluators had succeeded in establishing jointly six performance 
measures of services (see page 15). These measures became accepted by SIGMA as 
appropriate for monitoring and evaluating services and delivery. 

The major lessons learned from the different M&E efforts in so far as involving SIGMA 
were that: 1) it is better to accept a lesser objective with a partner and seek internalization 
of this than to insist on a broader aim, which is less achievable in the short-term, 2) it is 
important to address the priority concerns of stakeholders regarding performance 
monitoring, even if these have different value to the subjects of evaluation (i.e. SIGMA 
customers and SIGMA staff) and 3) it is difficult but not impossible to address confounding 
factors to demonstrate the process of performance monitoring, even when not perfect and 
produce results for program improvement. 

For the first lesson the issue concerned reaching agreement with SIGMA on defining 
performance measures for outputs and outcomes. Given the confounding factors 
discussed above, it is not surprising that SIGMA chose to agree on developing indicators 
limited to service delivery. A dialogue ensued to establish service indicators which SIGMA 
considered suitable. Among the more difficult measures upon which to reach agreement 
were cooperation with other donors, ownership (of the interventions and results) and the 
contribution the interventions provided to the continuation of reforms. However, once the 
six service measures had been accepted by SIGMA, data collection and analysis became 
easier and SIGMA was interested in and began to use M&E outputs produced in this 
framework. In the recommendations of the evaluator’s final report, PECAT urged SIGMA 
to re-examine the issue of program outcome performance measures. The evaluators also 
proposed consideration of three other indicators: 1) “catalyst” role of SIGMA, 2) 
diversification of financial support, and 3) substantive technical (sectoral) capacity to 
anticipate maturing/more complex client needs. 

With regard to the second lesson, PHARE, the main financier of the program, had several 
main priorities: to be able to have evidence which demonstrated accountability, cost- 
efficiency and coordination. These issues were important to SIGMA but of less priority 
than customer satisfaction and the internalization of program improvement. The evaluators 
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and SIGMA had to reach agreement on how best to balance the different interests and 
values. The six service indicators included PHARE concerns and a comparison of 
assistance programs with other donors was made to ensure SIGMA was “good value". 

Lesson three highlighted the partnership experience between SIGMA and the evaluators 
in so far as finding ways to address factors over which SIGMA had no control and provide 
tangible performance monitoring tools to manage the changing customer needs. As a 
result of the first two lessons, SIGMA and the evaluators came to make progress on how 
to treat the confounding factors of uncertainty, diversity, interdependence and instability. 
The monitoring reports identified short-term constraints and provided SIGMA opportunity 
to take corrective action before more serious problems developed. This meant the M&E 
reduced to a certain extent the uncertainty and instability faced by the program by allowing 
SIGMA to adjust or adapt its services. From the interviews and rapid appraisals, it was 
possible to obtain satisfaction rates which clearly demonstrated to SIGMA, stakeholders 
and customers that the program had an extraordinary appreciation. The analysis also 
helped identify the kinds of services customers considered of greatest benefit, thereby 
providing SIGMA information on trends, potential new frontiers and ways in which diversity 
and interdependence could be better addressed. Such feedback on performance 
increased SIGMA confidence as well as that of customers and stakeholders. 

By the end of the formal M&E period, SIGMA was better equipped to understand the data 
collection needs and options available for continuing to obtain such information. Staff had 
reviewed existing formats of designing and monitoring project interventions and developed 
new forms which internalized the use of the performance measurements into daily 
operations. 

C. Involvement of Customers and Stakeholders 

Under communist times, there was no monitoring and evaluation tradition. Performance 
monitoring and data collection served very different purposes. When assistance programs 
began to operate in the region, the concepts of "project", design, monitoring, 
implementation and evaluation had to be introduced to recipients. The transfer of this new 
management culture and related performance management principles, methods and 
instruments was seldom incorporated as an integral part in the assistance packages. 

SIGMA "customers" consisted of two groups: the country representatives forming the 
Liaison Group and the actual end-users of assistance. When the evaluators began to 
interview customers, much of the time had to be devoted to explaining the M&E function 
and the importance of frank, reliable data collection so that SIGMA could better provide 
relevant, quality services. The real beneficiaries from this kind of primary data collection 
and attempts at involving customers in the M&E work were the end-users of SIGMA 
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assistance, even if the exposure was limited. As the national representatives to the Liaison 
Group changed virtually every year from country to country. The evaluators could not 
safeguard meeting the representatives to provide elements for their learning or performing 
better the function of the Steering Committee of the program. The reports generated from 
the data collection provided at least a base for Committee members' understanding better 
the program and for taking decisions on policy matters. The evaluators in turn had no 
scope in their contract to further involve the end-users or the country representatives - an 
opportunity which could and should have been expected for follow-up after the formal M&E 
work ended. 

Finally, a significant lesson learned from the M&E experience relates to customers 
beginning to learn from each other. As more and more information became available to 
customers from SIGMA and the evaluators, the realization developed that needs, 
approaches, strategies, and solutions to given reform problems often had more in 
common with the CEE countries than with some in the West. As a result, the request for 
multi-country events and/or expertise from the region increased and sometimes even 
moved outside the program. For example, the Romanians invited at their own cost Poles 
for work in oversight (sector 5) and procurement (sector 3); Hungarians went to Slovenia 
on procurement work; and Latvians participated in study tours to various CEE countries. 
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IV. Concluding Remarks 

The SIGMA and PECAT partnership proved unique in the assistance experience to the 
CEE countries and valuable in contributing to the growing application and study of 
performance monitoring. In the broadest sense the work illustrates that an enabling 
environment for M&E requires senior management commitment and willingness to learn 
and that performance measures need to be defined for both service and program delivery. 
The need from the beginning for significant involvement of program customers in M&E and 
provision of adequate resources for M&E services and functions also emerged as 
important lessons. 

While these findings have been encountered in other contexts they provide additional 
evidence of their generalizability. 




Policy Education Centre on Assistance to Transition 



30 



33 



Monitoring and Evaluation of SIGMA 2 
CES Conference, Ottawa 



5 May 1997 



Selected Bibliography 



Buxell, Ingrid T, Ners, Krzysztof J. (1995) Assistance to Transition: Survey 1995. Warsaw: 
Policy Education Centre on Assistance to Transition, Institute for EastWest 
Studies. 

Buxell, Ingrid T., Ners, Krzysztof J. & Houwelingen, Jan W.A. (1996) Final Evaluation 
Report on the Evaluation of SIGMA (I) 2. Warsaw: IEWS Policy Education Centre 
on Assistance to Transition. 

Carter, N. (1991). Learning to Measure Performance: The Use of Indicators in 
Organizations: Public Administration 69. 

De Dreu, Karsten K. W. & van de Vliert, Evert. (1997). Conflict Management and 
Organizational Performance: Sage Publications, Inc. 

Epstein, P. D. (1992). Measuring the performance of public services. In M. Holzer (Ed.), 
Public Productivity Handbook, pp. 161-1 93. New York: Marcel Dekker, Inc. 

European Council. (1994). Presidency Conclusions, Meeting on 9-10 December 1994 in ' 
Essen. Brussels. 

Mayne, John & Zapico-Goni, Eduardo (Eds.) (1997). Monitoring Performance in the Public 
Sector. New Brunswick, USA: Transaction Publishers. 

Morgan, Gareth (1997). Images of Organization: Sage Publications, Inc. 

Nyhan, R. C. & Marlowe, H. A. (1995). Performance measurement in the public sector: 
Challenges and opportunities. Public Productivity and Management Review, 18(4), 
333-348 

Patton, M. Q. (1986). Utilization focused evaluation (2 nd ed.). Newbury Park, CA: Sage 
Publications, Inc. 

Rossi, Peter H. & Freeman, Howard E. (1993). “Strategies for Impact Assessment”. In * 
Evaluation a Systematic Approach. Newbury Oak, CA: Sage Publications. 

Wholey, J. S. (1983). Evaluation and Effective Public Management. Boston: Little, Brown 

Wholey, J. S„ Abramson, M. A. & Bellavita, C. (1986). Performance and Credibility: 
Developing Excellence in Public and Nonprofit Organizations. Boston: Lexington 
Books. 

Wholey, J. S. & Hatry, H. P. (1992). The case for performance monitoring. Public 
Administration Review, 52, 604-610. 



Policy Education Centre on Assistance to Transition 



31 



ANNEX 1 



EVOLUTION OF SIGMA 1 AND 2 



Annex 1: Evolution of SIGMA 1 and 2 



5 May 1997 



EVOLUTION OF SIGMA 1 AND 2 



For ease of reference, this Annex provides brief summaries.. of the Program Orientations 
of SIGMA 1 and 2. 



Program Orientation SIGMA 1 (11 March 1992) 

Mission Statement 

Support the development of efficient and effective public institutions which can sustain 
market economies, provide a base for democratic pluralist systems of governance and 
implement public policies. 

To achieve this, SIGMA would aim to: 

• build up central government capabilities to develop and coordinate policies, to 
manage the evolution of public institutions and the public service, to carry out resource 
management and control functions; 

• encourage central management agencies to promote reform in public bodies 
responsible for delivering public services in priority areas. 

It would also promote cooperation and networking, undertake studies, and facilitate sharing 
of experience on the transition, on a regional basis. 

Process 

• Missions to the 5 (now 6) recipient target countries to identify: 

- contextual situation 

- state of reform in public administration 

- key targets of transformation strategy 

- priority areas for SIGMA support 

- comparative advantages of SIGMA in relation to other support programs 

• Discussion within the SIGMA Liaison Group, based on mission findings, in order to 
determine content and strategy for SIGMA support activities 

• Implementation 
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Key Targets of Transformation Strategy 

Substantive reforms should aim to: 

a) strengthen policy-making capacities, improve quality of legislation, introduce 
implementation planning, improve the ability of governments' to direct resources to 
priorities while respecting budget limits; 

b) enhance the ability of the public and governments to guide and control administrative 
activity and keep a balance between the extent of administrative discretion and the 
instruments of control; 

c) ensure that a firm base of ‘rule of law’ institutions is established as soon as possible; 
and 

d) create a limited core of stable, institutionalized, competent, neutral professional public 
servants who have an ethical framework and a ‘public service mentality’. 

Tactically reforms should: 

e) use the principle of selective radicalism (concentrate change resources to bring about 
radical change in a few areas where interdependencies will force other parts of the 
system to respond); 

f) deliver improvements within politically relevant time-scales in politically relevant areas; 
and 

g) be mutually supportive and create a momentum for change which can continue through 
changes in leadership. 

Priority Areas for SIGMA Support 

i) Management of Policy-making; Strengthen the policy coordination and legislative 
review system 

ii) Expenditure Management; Strengthen the system for allocating and controlling public 
resources 

iii) Public Service; Create a neutral, competent, professional, stable public service 

iv) Administrative Control; Reinforce institutions exercising control over administrative 
actions. 

Condition Setting 

• generate sustained political attention for reform 

• avoid over politization of reform 

• elevate the transformation of public institutions to status of policy domain 

• create public administration- or central reform unit, located at the centre of 
Government 
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• develop wider understanding of and interest in civil society in order to provide both 
continuity and pressure for change 

• coordinate external support 

Methods of Work V- 

• traditional forms of TA (training, consultancy, seminars, etc.) 

• second SIGMA staff member and possibly other consultants, for longer periods, to key 
institutions (central reform unit) responsible for PAR and central policy coordination 

• responsive to fast-changing needs 

• support self-diagnosis as preliminary to reform 

- generate analytical profile of public sector with analysis of strategic directions for 
reform to be used as input for national reform programs and orient SIGMA support 

• work with NGOs to ensure continuity and create a climate of opinion and understanding 

• use of regional approaches 




Policy Education Centre on Assistance to Transition 



Page A3 



Annex 1: Evolution of SIGMA 1 and 2 



5 May 199 7 



Program Orientation SIGMA 2 (19 October 1995) 

Program Work Areas (Activities) 

1) Reform of Public Institutions : 

- Countries which are further ahead on macro-economic stabilization have identified 
institutional reform as the next priority. 

2) Management of Policy-Making 

- Organizational arrangements of policy-making need strengthening. Policy-making 
has to be made more cost-effective, user-friendly, transparent, participatory and 
less open to abuse. 

3) Expenditure Management 

- Reform of budgeting is necessary to reform of administration: without budget reform 
administrative reform might be blocked. 

4) Management of Public Service Personnel 

- The management of people has to be reformed to establish a professional, efficient, 
merit-based, impartial and non-corrupt public service, serving the collective 
government. 

5) Administrative Oversight 

- A major threat to new democratic systems is the re-emergence of old patterns of 
behavior - corruption, abuse of power, arbitrary decision-making, etc. 

6) Information Services 

- New aim for SIGMA 2 is to improve sharing of information on public administration. 

• Main theme: In search of good governance 

• Broader subject than SIGMA’s focus on core management systems. However, 
these systems are vital tools which democratic governments use to generate well- 
performing public functions. 

• “Modernization" rather than “reform”. 

• SIGMA's approach is based on OECD experience. OECD governments use the levers 
of core management systems - policy-making, personnel management, expenditure 
management and administrative oversight. In the transition countries these central 
(core) management systems have to be build up. 

• How can SIGMA contribute to the search for “good governance”? The program’s 
priorities are to help: 

• install effective public management systems; 

• improve the balance between managerial and legal reform, and strengthen 
capacities to implement reform; 
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• clarify relations between institutions of the State and levels of government from the 
perspective of central management systems; 

• orient administrative reform so that it contributes to achievement of economic, social 
and foreign policy goals; and 

• improve flows of information about administration, v-' ... 

• Current funding is about 60 percent of requested level. In order to cut areas of work 
rather than reduce the level of involvement, the program will focus on only three areas 
and the Information Services. 

• Of the core management systems, SIGMA’s priorities are for Management of Policy- 
Making (2), Management of Public Service Personnel (4), and Expenditure 
Management (3). 

Working Methods 

• SIGMA’s work will continue to be guided by the following values: 

• responsive to needs in countries as they develop; 

• use of high-quality experts; 

• exploratory approach, with no preferred models; and 

• sense of partnership 

• SIGMA will enhance the basic resource of SIGMA’s professional staff, use more 
experts from the transition countries, and ensure access to administrative law experts. 

• Where useful for practitioners, SIGMA will do more documentation-gap filling work for 
“shared consumption”. 

• SIGMA will devote more effort to marketing to ensure that outputs are used. Possible 
marketing methods: organizing briefings for Parliamentarians, or working with the' 
media. 

• Multi-country work will be less frequent and with a narrower focus. They may be 
designed around the interests of only a subset of countries. 

• SIGMA will set up an information service to ensure better knowledge of SIGMA itself 
and the outputs of its activities with the following four components: 

• newsletter on public administration and donor activity; 

• updated and extended country profiles; 

• documentary base for consultation; and 

• distribution service for SIGMA and PUMA material. 

• SIGMA will aim to upgrade its cost-efficiency by searching for best discounts for tickets, 
higher counterpart contributions, and payment for SIGMA services. 
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Collaboration with other donors 

• SIGMA was set up with the objective of helping countries use donor resources better 
and to foster collaboration. Mechanisms for doing this are: 

• The Liaison Group itself (brings together donors and recipients); 

• Information produced under SIGMA 1, especially the country profiles; 

• Help “package" projects for financing by other donors, and also help the beneficiary 
country to manage such projects. This service of SIGMA will be developed further; 
and 

• Participate in projects of other donors and have resources from other donors or 
multilateral organizations participate in SIGMA projects. 

Development of SIGMA Organization 

• SIGMA will be organized in such a way that each Activity will be implemented by a unit 
of one senior counselor and two or three advisers, plus support staff. The Information 
Services Activity will have one professional and one support staff. 

• SIGMA will maintain a country service function and will phase out in-country staff. 

Targets for the end of the Mandate 

• As its contribution to national reform programs, SIGMA should have achieved the 
following: 

• provided valued, cost-effective inputs to national reforms; 

• be recognized as a source of expertise by countries and donors; 

• consolidated a responsive and cost-effective mechanism for delivering services to 
CEEC's; 

• established a information service, linked to professional communities and 
institutions in CEEC and G-25 networks; and 

• because it has been judged a cost-effective support to countries in transition, 
generate stable, efficient and sufficient financing arrangements for SIGMA 3. 
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METHODOLOGICAL CONCERNS 



Baseline : While it can be considered that the “baseline” for SIGMA constituted the 
introduction of interventions at base “Zero,” the characteristics of this base varied 
extensively from country to country, sector to sector and even more so, from sub-sector 
to sub-sector. SIGMA 1 developed with the SIGMA Liaison Group on 11 March 1992 a 
Program Orientation document which among other matters surveyed the needs for and 
obstacles to administrative reform in the six countries of the program. Subsequently, 
SIGMA 1 began to establish a “conceptual base” for dialogue within each nation on specific 
problems of reform through the regional workshops. SIGMA Country Profiles further 
defined the broad public management context for the eleven SIGMA countries. This 
documentation provides the broad parameters of the “base” which SIGMA sought to 
change. 

Program Performance Measurements : When SIGMA 1 was designed and developed, it 
had no precedents or ready-made measurements according to which performance could 
be readily assessed. By virtue of its nature and the changing context reality, the program 
had to learn what to do as it went along. Under these circumstances, it was recognized by 
the SIGMA Liaison Group and SIGMA’s main funder, EU/PHARE, that only a demand drive 
approach over detailed programmatic design would work. 

Hence, PECAT, in partnership with SIGMA, developed service related performance 
measurements which could inform management on quality if not on program performance 
and reform change. 

Program Coverage and Causality : SIGMA is not the only provider of technical assistance 
for improving administrative reform. The contributions of other donors and national 
initiatives in this sector make these contributions “partial" program coverage efforts. This 
poses problems of causality and makes the M&E task so complex that it is realistically not 
possible to sort out which interventions have had which specific results. At the same time 
this complication is also a challenge to the evaluator. Therefore, it was decided to conduct 
two informal customer satisfaction appraisals and determine from clients the contribution 
SIGMA assistance had made. The appraisal results fell short of determining the specific 
overall program impact on public sector reforms by country and by sector. 
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Sample Methodology 



Sampling 

The following risk formula was adopted for selecting the sample randomly: 

ER = IHRx IORxSR 

Where the items in the formula have the following meaning: 

ER: Evaluation risk, the risk that the evaluation will come up with wrong conclusions 

IHR: Inherent risk, the risk associated with the type of “business’. For example, in general 

the risk in banking and financing are larger than in farming or sheep raisin. As a 
starting point a neutral position (50%) is acceptable, witch can be adjusted in 
following evaluations depending on the findings that provide a more in-depth 
clarification of the nature of the risk that are inherent for the type of environment. 
IOR: internal organization risk, the risk that the internal organization with it’s in build 
internal control mechanisms (written procedures, specialized control and audit 
bodies, management oversight, etc.) will or can not detect errors. 

SR: sample risk, the risk that a sample is not representative. This is a major criteria in 

the determination of sample size. The smaller a sample size, the higher the risk that 
the selected items are not representative. 

The values in the risk formula that were adopted are as follows. PECAT’s assessment of 
the inherent risk is neutral (50%). The assessment of the internal organization risk (based 
on PECAT’s work in reviewing SIGMA/OECD procedures and interviews) is 10%. With a 
maximum evaluation risk of 1%, this led to the conclusion that a sample risk of 20% is 
acceptable (or following the above formula: 1% 50% x 10 % x 20%). 

Sample Methodology 

The sample was drawn in two occasions. For the first sample drawing the selected 
methodology was the monetary unit sampling method. In this method, the universe are not 
the items under review, but their related monetary units. There is an explicit bias in that 
items with larger monetary units (or more costly projects) have a higher probability to be 
selected in the sample. The sampling software selects randomly a number of francs out 
of this universe. On the basis of the risk, a sample size of 1 1 was calculated, so 1 1 random 
francs were selected. Those selected francs were related to 1 1 projects. It appeared that 
3 francs were already selected, so that in fact 8 different projects were selected for in- 
depth evaluation. 
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For the second drawing, the methodology was slightly altered by taking into consideration 
the Sectoral proportions of indirect costs so as to ensure a more even Sectoral 
representation in the sample. The second sample consisted of 7 projects causing the total 
sample to be 1 5 projects. 

Sample Description 

The selection of the sample resulted in the following country and Sector distribution: The 
sample covers six out of eleven countries and five out of six Sectors. Together, the 15 
projects represent 23.3% of all direct costs allocated to SIGMA interventions. 
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